We investigate the thermodynamics of a four-dimensional charged black hole in a finite cavity in asymptotically flat and asymptotically de Sitter space. In each case, we find a Hawking-Page-like phase transition between a black hole and a thermal gas very much like the known transition in asymptotically anti-de Sitter space. For a ``supercooled'' black hole--a thermodynamically unstable black hole below the critical temperature for the Hawking-Page phase transition--the phase diagram has a line of first-order phase transitions that terminates in a second order point. For the asymptotically flat case, we calculate the critical exponents at the second order phase transition and find that they exactly match the known results for a charged black hole in anti-de Sitter space. We find strong evidence for similar phase transitions for the de Sitter black hole as well. Thus many of the thermodynamic features of charged anti-de Sitter black holes do not really depend on asymptotically anti-de Sitter boundary conditions; the thermodynamics of charged black holes is surprisingly universal.